Systems and methods for device positioning surface detection

ABSTRACT

Methods, hardware, and software establish surfaces for graphics processing by setting a position of the capture device fixed on or relative to that surface. A user may hold the device against the surface or through other means. Ambient setting, potentially including the surface itself, is visually captured at the position. Using that position the surface is established—readily known by position, size, shape, etc. within the surroundings. Stable positioning can be determined in any way, using any data. Motion of the device may be known via internal or external sensors, and stable surface touching can be detected with a lack of motion. A user may input when the device is steady. Abutment can be measured from visual input or impact sensors. All such input may determine and/or verify a stable position. Visual input from the stable position may thus be used to establish the surface for graphical processing.

BACKGROUND

FIG. 1 is a schematic of a related art user device 100 illustratingcomponents thereof that may permit creation of immersive content, suchas augmented reality (AR) 101. AR applications may use an overlay ofgraphical/added subject matter on live or recorded video or still imagespresented on device 100. For example, a user or application may positiona static graphic, text, or other visual element superimposed on theunderlying video or image. Augmented reality applications may alsoaugment underlying visual data by changing or replacing select elementsin the underlying environment and tracking those elements as if they areactually present in the underlying environment, or use the underlyingvisual data to position the augmented reality subject matter asdisplayed. For example, an underlying object may be recognized in avideo feed or image still of an environment, and the augmented realityapplication may superimpose additional elements on or apply coloration,designs, distortions, etc. to, or relative to, the object, so as tofurther the augmented reality effect that the object and surroundingenvironment actually includes such elements or characteristics.

To handle AR, device 100 may include a camera package 110 including alens, image sensor, and/or microphone, a computer processor 120,persistent and/or transient storage 130, external communications 140,display 180, AR configuration 170, and/or input device and input sensors185. Although elements are shown within a single device 100, it isunderstood that any element may be separate and connected throughappropriate communications such as an external bus for a peripheral orwired or wireless connection. Processor 120 may include one or morecomputer processors connected to and programmed or otherwise configuredto control the various elements device 100. Processor 120 may further beconfigured to create, transmit, and/or perform augmented reality 101 inaccordance with programming or input and may potentially include anassociated processor cache, transient memory, video buffer, etc.,configured or programmed to processes AR 101. For example, ARconfiguration 170 may include software and/or firmware that instructprocessor 120 how to create or display AR 101 received from anapplication or outside source. Processor 120 may also receive sensorinformation from sensors 185, e.g., touch or cursor information, andprocess the same as user interaction or input. Processor 120 may furtherexecute software or include configured hardware that allows forexecution of example methods discussed below.

Storage 130 may be a dedicated data storage drive or may be a partitionof a general data store in which augmented reality information, originor limitation information, application information, and/or deviceoperations and raw data can be saved. Storage 130 may be, for example,random access memory (RAM), read only memory (ROM), programmable readonly memory (PROM), erasable programmable read only memory (EPROM),electronically erasable programmable read only memory (EEPROM), flashmemory, a hard disk, a processor cache, optical media, and/or othercomputer readable media.

Camera 110 may include one or more lenses and/or apertures that may becontrolled by actuators that move the lenses and apertures amongdifferent positions to focus captured optical data. Similarly, camera110 may adjust focus digitally or in response to user input definingfocus locations in the scene being captured. Camera 110 may includeimage sensor elements such as a charge coupled device (CCD) array, aphotodiode array, or any other image sensing device that receives light,potentially via the lens, and generates image data in response to thereceived light. Camera 110 may include a light to aid in reflectionand/or focusing laser. Camera 110 may be further configured to obtain oradjust image information such as focus, zoom, white balance, exposure,saturation, and/or other image functions. Camera 110 and/or processor120 may be further configured with one or more video codecs or otherimage processing software or drivers to capture, process, and storeexternal independent media such as actual video from the environment aswell as augmented reality. An included microphone may be any auditorytransmission and/or reception device capable of audio pickup and/orplayback.

Display 180 may be a screen, viewfinder, monitor, projector, glassesfront, or any other device capable of visually displaying visualaugmented reality 101. For example, display 180 may be a touchscreen ona smartphone like iOS or Android devices or on a tablet like an iPad orSurface, or display may be an LCD monitor or projector, for example.Sensors 185 provide input information, potentially as a part of display180 and/or as a separate component. For example, if display 180 is atouchscreen, sensors may be embedded multi- or single-touch capacitivesensors capable of detecting finger or stylus touch, pressure, movement,etc., with respect to display 180. Or for example, sensors 185 may be anaccelerometer or magnetized compass with associated hardware or softwarecapable of determining device orientation and/or movement. Or forexample, sensors 185 may be a button or an external mouse or joystickand associated hardware or software capable of controlling anddetermining cursor position and/or activation with respect to display180 during operation of device 100. Sensors 185 are connected toprocessor 120 and can deliver sensed input information to processor 120with respect to display 180, including cursor or contact position,duration, numerosity, pressure, movement speed, etc.

Device 100 may further include a communications port 140 for externalwired or wireless communication. For example, communications port 140may be an antenna configured to transmit and receive on CDMA bands, aWi-Fi antenna, a near field communications transmitter/receiver, a GPSreceiver, an external serial port or external disk drive, etc. Processor120 may provide data from storage 130, input data from camera 110,sensors 185, etc., to external devices through communications port 140,as well as receive application and/or augmented reality and otherinformation from providers through port 140. Further, communicationsport 140 may function as another input source for sensors 185.

Device 100 may be mobile, such as a laptop, smartphone, wearable,dashcam, camcorder, GPS device, etc., or device 100 may be relativelyimmobile, such as a desktop computer, wide-area network, fixed securitycamera, etc. Although networked elements and functionalities of device100 are shown in FIG. 1 as individual components with specific groupingsand subcomponents, it is understood that these elements may beco-located in a single device having adequately differentiated datastorage and/or file systems and processing configurations.Alternatively, the elements shown in FIG. 1 may be remote and plural,with functionality shared across several pieces of hardware, eachcommunicatively connected at adequate speeds to provide necessary datatransfer and analysis, if, for example, more resources or betterlogistics are available in distinct locations.

SUMMARY

Example methods and embodiment devices capture a surface, like acountertop, ceiling, or wall, for tracking in graphics processing,including immersive processing like AR or VR, by assuming ahigh-fidelity position with respect to that surface. For example, amobile device or computer may instruct a user to hold a camera and/orthe device against the surface, or may simply detect when such aposition is held. With the device at a known, relatively constantposition, the surroundings of the surface, potentially including thesurface itself, are visually captured at the position. Then, using thatknown position, the surface position, size, shape, and othercharacteristics within the surroundings can be accurately determined, orestablished, for use in graphics processing. For example, theestablished position can be output to an AR application that maymodify/augment and track the surface in the surroundings as displayed onthe device as AR.

Example methods and devices can determine stable positioning against thesurface in several different ways and with several different types ofinput. For example, motion of the device may be known through anaccelerometer, level, GPS receiver, compass, etc.; a lack of motion fora threshold period of time, or an abrupt stopping of motion followingmovement, may indicate stable abutment with the surface. Similarly, auser may be instructed to put the device against the surface and touchan on-screen button or give other input when the device is steadilypositioned. Yet further, visual input from the device's camera thatdetect motion, pressure sensors on the device that detect its beingpressed to the surface, a lack of any other activity on the device for athreshold amount of time, etc. may all indicate rest on the surface. Anyof these metrics may be used alone or in combination, potentially inserial or looped verification fashion, to ensure a stable position wherevisual input will establish the surface based on that stable position.

BRIEF DESCRIPTIONS OF THE DRAWINGS

Example embodiments will become more apparent by describing, in detail,the attached drawings, wherein like elements are represented by likereference numerals, which are given by way of illustration only and thusdo not limit the example embodiments herein.

FIG. 1 is an illustration of a related art device useable in graphicsprocessing.

FIG. 2 is an example embodiment device configured to execute examplemethods.

FIG. 3 is a screen flow illustrating actions and device states in anexample method.

DETAILED DESCRIPTION

Because this is a patent document, general broad rules of constructionshould be applied when reading it. Everything described and shown inthis document is an example of subject matter falling within the scopeof the claims, appended below. Any specific structural and functionaldetails disclosed herein are merely for purposes of describing how tomake and use examples. Several different embodiments and methods notspecifically disclosed herein may fall within the claim scope; as such,the claims may be embodied in many alternate forms and should not beconstrued as limited to only examples set forth herein.

Modifiers “first,” “second,” “another,” etc. may be used herein todescribe various items, but they do not confine modified items to anyorder. These terms are used only to distinguish one element fromanother; where there are “second” or higher ordinals, there merely mustbe that many number of elements, without necessarily any difference orother relationship. For example, a first element could be termed asecond element, and, similarly, a second element could be termed a firstelement unless an order or difference is separately stated. In listingitems, the conjunction “and/or” includes all combinations of one or moreof the associated listed items. The use of “etc.” is defined as “etcetera” and indicates the inclusion of all other elements belonging tothe same group of the preceding items, in any “and/or” combination(s).

When an element is related, such as by being “connected,” “coupled,”“mated,” “attached,” “fixed,” etc., to another element, it can bedirectly connected to the other element, or intervening elements may bepresent. In contrast, when an element is referred to as being “directlyconnected,” “directly coupled,” etc. to another element, there are nointervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(e.g., “between” versus “directly between,” “adjacent” versus “directlyadjacent,” etc.). Similarly, a term such as “communicatively connected”includes all variations of information exchange and routing between twodevices, including intermediary devices, networks, etc., connectedwirelessly or not.

As used herein, singular forms like “a,” “an,” and the are intended toinclude both the singular and plural forms, unless the languageexplicitly indicates otherwise. Indefinite articles like “a” and “an”introduce or refer to any modified term, both previously-introduced andnot, while definite articles like “the” refer to the samepreviously-introduced term. Possessive terms like “comprises,”“includes,” “has,” or “with” when used herein, specify the presence ofstated features, characteristics, steps, operations, elements, and/orcomponents, but do not themselves preclude the presence or addition ofone or more other features, characteristics, steps, operations,elements, components, and/or groups thereof. Rather, exclusive modifierslike “only” or “singular” may preclude presence or addition of othersubject matter in modified terms.

The structures and operations discussed below may occur out of the orderdescribed and/or noted in the figures. For example, two operationsand/or figures shown in succession may in fact be executed concurrentlyor may sometimes be executed in the reverse order, depending upon thefunctionality/acts involved. Similarly, individual operations withinexample methods described below may be executed repetitively,individually or sequentially, so as to provide looping or other seriesof operations aside from single operations described below. It should bepresumed that any embodiment or method having features and functionalitydescribed below, in any workable combination, falls within the scope ofexample embodiments.

The inventors have recognized that computers rendering immersiveexperiences or other image processing, including AR, often need todetermine surfaces in an environment for modelling quickly andaccurately—for example, seamlessly finding real-world horizontal andvertical planes on which to place or root virtual objects. Visual inputalone, such as through a camera, is typically used to identify surfacesand visual features thereon for image analysis and modification,including positioning, size, relative features, and orientation. Theinventors have recognized that such input often fails when surfaces areclear like glass, reflective like a mirror or glossy page, orfeatureless like a white wall. Detection is challenging and error pronebecause of the lack of visually-mappable features or characteristicsthat computer vision systems need to establish the presence, size, andorientation of the surface, e.g., to establish the surface in AR.Surface recognition and subsequent manipulation may thus be impossible,or cumbersome in requiring additional actions by a user such as taggingthe surface, changing lighting, or inputting physical characteristics ofthe surface, before it can be established. The inventors have developedexample embodiments and methods described below to address these andother problems recognized by the Inventors with unique solutions enabledby example embodiments.

The present invention is devices, software as stored or executed ontangible computer-readable media, and methods for inputting surfacesbased on stable device position with respect to the same. In contrast tothe present invention, the few example embodiments and example methodsdiscussed below illustrate just a subset of the variety of differentconfigurations that can be used as and/or in connection with the presentinvention.

FIG. 2 is an illustration of an example embodiment device 200 configuredto determine surface positioning and potentially execute examplemethods. For example, device 200 may establish a plane like afeatureless wall or floor for use in AR. As shown in FIG. 2, device 200may include several elements from related device 100 of FIG. 1. Exampleembodiment device 200 includes at least one sensor 285 outputting datauseable to determine surface location and/or orientation. For example,sensor 285 may include a compass, magnetometer, accelerometer, level,gyroscope, directional antenna, pressure detector or touch input, and/orarrays of the same capable of determining movement, rest state,orientation with respect to compass directions and/or gravity, globalposition, and/or user input of verification or device position.

Example embodiment device 200 is further configured to determine andutilize surface positioning from visual input of camera 110 as well asother input. Such configuration may be software, firmware, or hardwareexecuting image processing 270. Image processing 270 may include, forexample, surface and/or edge detection software or drivers. Imageprocessing 270 may include AR or VR functionality, including AR toolkitssuch as Spark AR, ARKit, and/or ARCore, whose documentation arerespectively available atsparkar.facebook.com/ar-studio/learn/documentation/reference/scripting/summary,developer.apple.com/documentation/arkit, anddevelopers.google.com/ar/develop/ios/cloud-anchors/overview,incorporated by reference herein in their entireties. Image processing270 may include separate image processing software stored in transientor permanent memory and executed by processor 120, its own GPU, and/orbe a functionality of processor 120 itself, such as through properprogramming, for example.

Sensor 285 may directly provide information to image processing 270and/or processor 120 to allow determination of planes and surfaces fromenvironmental imagery captured by camera 110. Image processing 270and/or processor 120 further use such determined planes or surfaces inpresentation of AR 101 on display 180. For example, an identifiedsurface may be recolored or uniquely filtered and tracked to anunderlying real surface in a visual environment captured by camera 110,such altered surface being performed with display 180 as AR 101. Camera110, sensors 285, display 180, processors 120, and image processing 270may all be collocated, such as in a mobile device, or remote. Forexample, input from camera 110 and/or sensors 285 may be from places farapart in space and time and potentially saved, with processor 120 and/orimage processing 270 executing example methods with them at yetdifferent locations or later times.

FIG. 3 illustrates an example method as a screen flow on an exampleembodiment device, which could be example embodiment device 200 fromFIG. 2, as a smartphone or tablet. As seen in FIG. 3, in S200, animage/video capture or AR mode may be activated on the device. This mayinclude activating a camera, selecting a native option, starting an ARapp, etc., that allows visual capture, such as by camera 110 in device200 of FIG. 2, of the surrounding environment. In S200, world trackingfor immersive addition, such as AR or VR, may be established by movingthe device and camera about the environment or otherwise inputtingenvironmental characteristics and relative positioning to establishworld tracking.

In S210, immersive setup is completed by checking that world tracking isestablished and/or accurate. If world tracking is established in S210,example method may proceed to S220. If world tracking is not establishedin S210, additional instruction may be given to a user or additionalanalysis applied to visual input by the image processor in S215. Theseactions of establishing accurate immersion are optional and may beconducted by native or specific AR applications, such as ARKit, AR Core,etc. incorporated by reference above, in, for example an image processor270 as shown in FIG. 2. It is understood that preparation of a devicefor immersive content and establishing world tracking, like for AR orVR, in S200 and S210 may be partial, incomplete, or omitted in examplemethods.

In S220, the processor, potentially through the device, requests theuser to place the device and/or camera at a known position with respectto a surface of interest. For example, a user may input an “identifywall” option in the device, and the device may instruct the user to“place the bottom edge of the device against the wall” in S220. Or, forexample, image processing software, such as that in image processor 270of example device 200 in FIG. 2, may detect a surface in visual input tothe device and then display an icon or external light indicating aparticular side or edge of the device should be placed on the surface.The processor may identify the detected surface with highlighting, text,or arrow, for example, or the user may input a detected surface, such asby clicking or pressing on the surface on a touchscreen. Alternatively,no surface may be detected by or input into the device in S220, in whichcase the device may simply await stabilization on a surface. Theindication or request for surface abutment may be delivered in any modeor format, potentially on or outside the device, in S220.

In S230, the processor determines if the device, or at least the visualcapture device, is stabilized on the surface. This may be done inseveral ways. For example, a user may select an option or give otherinput indicating stabilization on the surface. Or, for example, anaccelerometer or gyroscope may detect device movement in a directionfollowed by abrupt stopping of that movement without resumption,indicating the device is stabilized on the surface. Or, for example, abutton or pressure sensor on an edge of the device may detect newcontact with a surface without removal. Or, for example, a magnetometeror directional antenna may detect movement of the device with respect togeomagnetic north or a known signal emitter followed by no furthermovement in the previous manner, indicating the device has stabilizedagainst the surface.

The stable abutment on the surface may be verified and/or corrected inS230. If stabilization does not pass a threshold and/or has not beenindependently verified, example methods may proceed to S235. Forexample, in S230, a user may input a verification of stabilization, suchas by pressing a confirm button on the device or otherwise inputtingstabilization. In this instance, only upon receipt of such input mayexample methods proceed from S230 to S240; otherwise, the instruction inS220 may remain, awaiting user verification in S230. Or, for example,the device may use any number of sensors, including an accelerometer,magnetometer, pressure sensor, etc., discussed above, to determine thatthe device has remained stable for a threshold period of time, and, ifnot, proceed to re-measure in S235 until stabilization is determined inS230. Similarly, the device may use the camera itself to detect motionthrough changing visuals, or use the camera to verify plane or surfacepresence in a steady manner within the captured environment.

Of course, any combination of user input, sensor feedback, and/or visualcapture can be used as a filter in S235 to determine stabilization. Forexample, in S230 if a first sensor, for example, a level sensor,determines constant device level for several seconds, visual input maythen be checked in S235. If the visual identification shows movement orunsteady phone orientation, then the level sensor threshold may be resetand check again in S220 and S230. If the visual identification shows astable surface, then this may be considered verification ofstabilization in S230, and example methods may proceed to S240. Ofcourse, an additional or alternative user input check may further beused in S235. Similarly, thresholds and tolerances on sensor input forstabilization may be lowered or increased based on visual confirmationor not of stabilization in S235 and/or user verification. As such, userinput, sensor input, and visual input may interplay in S230 andpotentially in a looping manner through S235 while inputs are received,before stabilization is confirmed and methods proceed to S240.

In S240, a plane or other surface is established positionally based onthe stabilization in S230. By knowing that the device, or at least thecamera capturing the visual environment, is stable and co-located withthe surface to be established, the surface and other relative featuresin that environment may be assessed and properly tracked, regardless ofthe quality or lack of ideal visual input. For example, even if a wallcannot be detected by AR programming in visual input because it isfeatureless or reflective, in S240, the wall may be presumed in a stableposition with respect to the camera, and other detectable features maybe presumed stable. These presumed stable elements may then used toestablish or root the wall.

Similarly, in S240, additional input, potentially input gathered in S230and/or S235, may be used to establish the surface in S240. For example,if a particular device edge or front/back is specified in S220, then thesurface identified in S240 may be presumed stable and extending fromthat device portion, or at least oriented with regard to the camera in aknown manner from that device portion, in the visual capture. Similarly,device orientation with gravity as determined from sensors may indicatewhether a surface is horizontal or vertical, such as floors/ceilings orwalls.

Once the surface is established in S240, it is output to the ARprogramming for use in augmentation or other rendering. This may includeidentifying and/or tagging the surface in the AR format in thosestandards incorporated above, or otherwise describing or transmittingthe surface to another rendering/graphical processing application.Similarly, in S250, the surface determined in S200-S240 may be storedlocally or remotely and later accessed for use in AR or other graphicalapplications, potentially by other devices than those establishing thesurface.

As seen, in one example method from end-to-end, a user may activate ARon their mobile device, such as a tablet or smartphone, be instructed toplace the device against a wall to be modified in AR, place the deviceagainst that wall, and have the device automatically identify the wallbased on that positioning and establish that wall surface as an ARelement, accurately tracked and modified in AR. In this way, a user mayquickly and reliably input or enter the wall into the AR program, whichcould be a visual program that modifies the wall with augmented elementslike object placement on the wall, wall recoloring to simulate painting,simulated wall removal or room renovation, etc. The user may never haveto otherwise select the wall or input additional data to ensure the wallis properly tracked in AR, and it may not matter that the wall is blank,glossy, or nearly transparent like glass, in such an example, becausethe device's stable position can be used to root the wall in thesurroundings.

Given the variety of example functions described herein, exampleembodiment devices and methods of establishing surfaces may bestructured in a variety of ways to provide desired functionality. Indevices, other divisions and/or omissions of structures andfunctionalities among any number of separate visual capture devices,modules, processors, and/or servers are useable, including execution ona single machine or among distant, exclusive servers and processors.Example methods may include user authentication, data verification,privacy controls, and/or content screening. For example, in examplemethods, data may be encrypted and not retained at one or all points inexample methods, such that there may be no discoverable record ofaugmented reality, surrounding environment, and/or surface information.Example methods may take advantage of a user login model requiring userauthentication with a password over a secured connection and/or usingoperating-system-native security control and verification oncommunications devices, to ensure only verified, permitted human usersaccess example methods and potentially user accounts.

Some example methods being described here, it is understood that one ormore example methods may be used in combination and/or repetitively toproduce multiple options and functionalities for users of communicationsdevices. Example methods may be performed through proper computerprogramming or hardware configuring of networks and communicationsdevices to receive and perform and act in accordance with examplemethods, at any number of different processor-based devices that arecommunicatively connected. Similarly, example methods may be embodied onnon-transitory computer-readable media that directly instruct computerprocessors to execute example methods and/or, through installation inmemory operable in conjunction with a processor and user interface,configure general-purpose computers having the same into specificcommunications machines that execute example methods.

It will be appreciated by one skilled in the art that exampleembodiments may be varied through routine experimentation and withoutfurther inventive activity. For example, although a plane, like a flatwall, may be established for AR in some examples, it is understood thatother surface shaped and orientations may also be established, for ARand other image processing or analysis, through device placement inexamples. Variations are not to be regarded as departure from the spiritand scope of the exemplary embodiments, and all such modifications aswould be obvious to one skilled in the art are intended to be includedwithin the scope of the following claims.

What is claimed is:
 1. A mobile device configured to establish a surfacefor graphical processing, the device comprising: a display; a camera;and a computer processor configured to, instruct, through the display, auser to stabilize the camera against the surface to be established,determine that the camera is in a position stabilized against thesurface, capture, with the camera, environment at the position, andestablish the surface within the environment using the position.
 2. Themobile device of claim 1, wherein the computer processor is furtherconfigured to, augment the established surface in the environment asaugmented reality.
 3. The mobile device of claim 2, wherein the computerprocessor is further configured to, track, with the camera, theenvironment for the augmented reality before the instructing, whereinthe augmenting tracks the established surface in the environment asaugmented reality.
 4. The mobile device of claim 1, wherein the surfaceis a featureless, reflective, or substantially transparent plane, andwherein the environment captured by the camera includes the plane. 5.The mobile device of claim 1, wherein the position is a position of themobile device directly contacting the surface.
 6. The mobile device ofclaim 1, wherein the determining includes sensing motion of the camerafollowed by a lack of motion indicating the camera is stabilized on thesurface.
 7. The mobile device of claim 1, further comprising: anaccelerometer, wherein the determining uses at least one of user input,the accelerometer, and visual input from the camera to determine thatthe device is in the position stabilized against the surface.
 8. Amethod of establishing a surface for graphical processing, the methodcomprising: instructing, with a display, a user to stabilize a cameraagainst the surface to be established; determining, with a computerprocessor, that the camera is in a position stabilized against thesurface; capturing, with the camera, environment at the position; andestablishing, with the computer processor, the surface within theenvironment using the position.
 9. The method of claim 8, furthercomprising: augmenting, with the computer processor, the establishedsurface in the environment as augmented reality.
 10. The method of claim9, further comprising: tracking, with the computer processor and thecamera, the environment for the augmented reality before theinstructing, wherein the augmenting tracks the established surface inthe environment as augmented reality.
 11. The method of claim 8, whereinthe surface is a featureless, reflective, or substantially transparentplane, and wherein the environment captured by the camera includes theplane.
 12. The method of claim 8, wherein the display, computerprocessor, and camera are all co-located within a mobile device.
 13. Themethod of claim 12, wherein the position is a position of the mobiledevice directly contacting the surface.
 14. The method of claim 8,wherein the determining includes sensing motion of the camera followedby a lack of motion indicating the camera is stabilized on the surface.15. The method of claim 8, wherein the determining uses at least one ofuser input, an accelerometer, and visual input from the camera todetermine that the camera is in the position stabilized against thesurface.
 16. A method of establishing a wall for recoloring in augmentedreality, the method comprising, establishing world tracking through anaugmented reality application on a device having a camera, a display,and a processor; determining, with the processor, that the device hasbeen stabilized on the wall; capturing, with the camera, an environmentfrom the wall; and establishing the wall as an augmented reality elementin the environment shown on the display through the augmented realityapplication.
 17. The method of claim 16, further comprising: recoloringthe wall as the augmented reality element in the environment shown onthe display.
 18. The method of claim 16, further comprising:instructing, through the display, the user to stabilize the device onthe wall.
 19. The method of claim 16, wherein the determining includessensing motion of the device followed by a lack of motion indicating thedevice is stabilized on the wall.
 20. The method of claim 16, whereinthe determining uses at least one of user input, an accelerometer, andvisual input from the camera to determine that the device is in theposition stabilized against the wall.